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ABSTRACT 

The  decomposition  of  an  image  into  its  primitive  components, 
such  as  cartoon  plus  texture,  is  a  fundamental  problem  in  im¬ 
age  processing.  In  [11,  16],  the  authors  proposed  a  technique  to 
achieve  this  decomposition  into  structure  and  texture.  These  two 
components  are  competing  ones,  and  their  proposed  model  has  a 
critical  parameter  that  controls  this  decomposition.  In  this  paper 
we  show  how  to  automatically  select  this  parameter,  and  demon¬ 
strate  with  examples  the  importance  of  this  optimal  selection. 

Keywords:  Image  decomposition,  structure,  texture,  bounded  vari¬ 
ation,  parameter  selection,  inpainting. 

1.  INTRODUCTION 

Natural  images  usually  contain  two  main  components,  which  are 
broadly  classified  as  structure  and  texture.  Structure  is  mainly  the 
sharp  edges  in  the  image,  which  generally  separate  different  ob¬ 
jects,  or  are  caused  by  differences  in  shading.  Texture,  on  the  other 
hand,  is  in  general  a  repetitive  pattern,  or  “oscillations.”  The  de¬ 
composition  of  an  image  into  two  parts,  where  one  image  contains 
the  structure  and  the  other  image  contains  the  texture,  is  important 
in  a  number  of  different  areas,  such  as  copying  just  the  texture  in 
one  image  onto  another  image,  segmentation  of  an  image  based 
upon  the  texture  or  DC  gray-values,  etc.  This  decomposition  has 
been  shown  in  [6]  to  be  fundamental  for  image  inpainting,  the  art 
of  modifying  an  image  in  a  non-detectable  form  [2,  3,  5,  7,  10]. 

Following  work  by  Meyer  [11],  Vese  and  Osher  proposed  in  [16] 
an  algorithm  to  achieve  this  decomposition,  this  being  the  tech¬ 
nique  exploited  in  [6]  for  image  inpainting  (see  also  [1,9,  12,  14] 
for  other  related  decomposition  approaches).  As  we  will  see  bel¬ 
low,  there  is  a  critical  parameter  in  this  decomposition  that  controls 
the  competition  between  the  structure  and  the  texture  components. 
This  parameter  was  left  to  the  user  to  select,  in  the  original  for¬ 
mulation.  Our  goal  in  this  paper  is  to  simplify  the  decomposition, 
and  thereby  the  above  mentioned  applications,  by  designing  a  very 
simple  procedure  for  the  automatic  selection  of  this  critical  param¬ 
eter.  The  devised  method  has  been  motivated  by  the  approach  used 
in  [13]  (see  also  [4]  for  automatic  local  selections  of  critical  param¬ 
eters  for  this  TV  model),  and  automatically  finds  the  parameter  for 
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the  Office  of  Naval  Research,  the  National  Science  Foundation,  the  Presi¬ 
dential  Early  Career  Awards  for  Scientists  and  Engineers  (PECASE),  and 
National  Science  Foundation  CAREER  Award. 


the  best  possible  decomposition  in  terms  of  a  given  penalty  equa¬ 
tion.  We  should  note  that  this  is  in  sharp  contrast  with  multiscale 
approaches  such  as  those  reported  in  [15],  where  the  authors  opt 
for  selecting  a  set  of  parameters,  thereby  producing  a  set  of  de¬ 
compositions  instead  of  just  an  optimal  one  as  suggested  here. 

In  the  following  sections,  we  describe  our  proposed  technique  and 
illustrate  its  virtues  by  real  examples. 

2.  IMAGE  DECOMPOSITION:  BACKGROUND  AND 
OUR  APPROACH 

In  this  section,  we  will  first  throw  some  light  on  the  image  decom¬ 
position  approach  proposed  in  [16]  and  [11].  The  main  ingredients 
of  this  decomposition  are  the  total  variation  minimization  of  [13] 
for  image  denoising  and  restoration,  and  the  space  of  oscillating 
functions  introduced  in  [11]  to  model  texture  or  noise.  We  should 
note  that  this  decomposition  is  not  just  a  low-pass/high-pass  one, 
since  both  the  structure  and  texture  images  contain  high  and  low 
frequencies,  but  of  different  type. 

Let  I  :  ^  be  a  given  observed  image,  I  G  I  could 

be  just  a  noisy  version  of  a  true  underlying  image  u,  or  could  be  a 
textured  image,  u  then  being  a  simple  sketchy  approximation  or  a 
cartoon  image  of  I  (with  sharp  edges).  A  simple  relation  between 
u  and  I  can  be  expressed  by  a  linear  model,  introducing  another 
function  u,  such  that  I{x,y)  =  u{x,y)  +  v{x,y).  In  [13],  the 
problem  of  reconstructing  u  from  I  is  posed  as  a  minimization 
problem  in  the  space  of  functions  of  bounded  variation  BV{R^), 
[8],  allowing  for  edges: 

mJ^|F(M)  =  j \Vu\^-\\\v\\l2,  I  =  u  +  vy  (1) 

where  A  >  0  is  a  tuning  parameter.  The  second  term  in  the  energy 
is  a  fidelity  term,  while  the  first  term  is  a  regularizing  term,  to 
remove  noise  or  small  details,  while  keeping  important  features 
and  sharp  edges.  The  selection  of  the  optimal  parameter  A  for  this 
model  is  dealt  with  in  [4,  13]. 

In  [11],  Meyer  proved  that  for  small  A  the  model  will  remove  the 
texture.  To  extract  both  the  u  G  BV  (a  piecewise  constant  or 
cartoon  representation  of  the  image),  and  the  v  component  as  an 
oscillating  function  (texture  or  noise)  from  /,  Meyer  proposed  the 
use  of  a  different  space  of  generalized  functions,  which  is  in  some 
sense  the  dual  of  the  BV  space  (and  therefore,  contains  oscilla¬ 
tions). 

Inspired  by  Meyer,  in  [16],  Vese  and  Osher  devised  and  solved  a 
variant  of  his  model,  making  use  of  partial  differential  equations. 


The  following  minimization  problem  is  the  one  proposed  in  [16]: 
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where  A,  >  0  are  tuning  parameters,  andp  ^  oo.  The  first  term 
ensures  that  the  cartoon  image  u  G  BV{R^),  the  second  term 
ensures  that  /  ^  u  +  div(^i,^2)  {{91^92)  are  the  two  auxiliary 
functions  used  by  Meyer  to  define  the  new  norm  for  oscillations), 
while  the  third  term  is  a  penalty  on  the  norm  of  u  =  div(^i ,  ^2). 
This  follows  the  space  for  oscillations  proposed  in  [11],  where  the 
texture  image  v  is  given  by  div(^i  ^92),  with  the  pair  {91 , 92)  hav¬ 
ing  minimal  norm. 

Forp  =  1,  the  corresponding  Euler-Lagrange  equations  are  [16] 
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As  can  be  seen  from  the  examples  in  [16] ,  the  minimization  model 
(2)  allows  to  extract  from  a  given  real  textured  image  I  the  com¬ 
ponents  u  and  V,  such  that  u  is  a  sketchy  (cartoon)  approximation 
of  /,  and  V  =  div{9i,  92)  represents  the  texture  or  the  noise  (note 
once  again  that  this  is  not  just  a  low/high  frequency  decomposi¬ 
tion).  For  some  theoretical  results  and  the  detailed  semi-implicit 
numerical  implementation  of  the  above  Euler-Lagrange  equations, 
see  [16]. 

Our  goal  in  this  paper  is  to  develop  a  simple  and  fast  method  to  au¬ 
tomate  the  selection  of  the  tuning  parameter  A  so  that  we  minimize 
the  energy  in  (2).  This  parameter  controls  the  competition  between 
the  structure  part  u  and  the  texture  part  v,  and  if  wrongly  selected, 
it  will  leave  texture  in  u  and/or  strong  boundaries  on  u,  thereby  not 
achieving  the  main  goal  of  the  decomposition  and  making  subse¬ 
quent  uses  of  it  more  difficult. 

To  achieve  this  goal,  we  use  a  method  similar  to  the  gradient  pro¬ 
jection  method  used  in  [13].  We  first  make  an  approximation  re¬ 
sulting  from  the  model  proposed  in  [13],  that  at  equilibrium  state. 


gives  us  a  simple  expression  for  computing  Xnew,  for  the  next  it¬ 
eration  of  the  above  Euler-Lagrange  equations.  Although  the  as¬ 
sumptions  mentioned  above  do  not  need  to  be  made,  and  a  simi¬ 
lar  computation  can  be  done  without  assuming  the  approximation 
for  92  and  without  neglecting  the  second  term  in  the  varia¬ 
tional  formulation,  making  these  assumptions  simplifies  the  com¬ 
putation,  and  experimentally  we  haven’t  observed  any  significant 
improvement  when  the  real  values  of  these  terms  are  considered.  ^ 


3.  EXPERIMENTAL  RESULTS 

The  results  shown  next  have  be^  generated  using  the  ‘Image  Pro¬ 
cessing  Toolbox’  in  MATLAB^.  Since  there  is  just  an  addition 
of  one  simple  step  for  getting  the  new  A  at  every  iteration,  there  is 
very  little  overhead  and  the  decomposition  takes  the  same  amount 
of  time  as  in  the  case  of  [16].  Figure  1  shows  the  decomposition 
of  the  original  image  (top),  and  the  corresponding  cartoon  (u)  and 
texture  (v)  images,  when  an  arbitrary  value  of  A  was  used  for  the 
decomposition.  Notice  that  there  is  still  a  clear  component  of  the 
brick  texture  left  in  the  cartoon  image  which  was  not  transferred 
completely  to  the  texture  image.  The  use  of  equation  (7)  leads  to  a 
much  smoother  and  piecewise  constant  cartoon  image  as  described 
in  Figure  2.  The  optimal  lambda  used  also  gives  a  much  smaller 
value  for  the  energy  in  Equation  (2),  as  shown  in  the  plot  on  the 
left.  In  Figure  3  we  show  that  arbitrary  selections  of  A  can  affect 
not  only  the  u  image  as  in  Figure  1,  but  the  v  image  as  well.  The 
slight  light  variation  in  the  original  image  is  passed-on  to  the  v 
(texture)  image  with  arbitrary  A,  while  this  is  eliminated  with  our 
algorithm.  Lastly,  in  Fig.4,  we  describe  the  importance  of  auto¬ 
matic  selection  of  the  best  A  by  an  illustrative  example  from  the 
field  of  inpainting.  The  left  figure  was  decomposed  and  inpainted 
as  in  [6],  by  using  an  arbitrary  A  (center)  and  with  the  optimal  A 
(right).  The  region  pointed  to  by  the  arrow  shows  the  discrepancies 
in  inpainting  caused  due  to  a  sub-optimal  decomposition. 


4.  CONCLUDING  REMARKS 


9iix,y)  +  gl{x,y)  =  (6) 

see  [16]. 

At  every  iteration  of  implementing  the  above  Euler-Lagrange 
equations  (3,4,5),  we  assume  that  we  have  reached  equilibrium. 
Hence,  the  second  term  contributing  to  the  energy  in  (2)  is  close 
to  zero.  Let  us  call  the  new  (minimizing)  values  of  u,  and  92 
by  the  names  Unew ,  9inew  Q^new  •  Suppose  that  the  new  value 
of  A  (say,  Xnew)  minimizes  the  energy.  Then,  in  a  method  similar 
to  the  gradient  projection  method,  we  equate  the  energy  calculated 
from  the  previous  values  with  that  calculated  from  the  new  values 
which  are  supposed  to  attain  equilibrium.  Thus, 


In  this  note  we  have  shown  how  to  automatically  compute  the 
critical  parameter  in  the  image  decomposition  model  proposed  in 
[11,  16].  The  technique  is  very  simple  and  adds  virtually  no  com¬ 
putational  cost  to  the  overall  decomposition  procedure.  We  should 
note  that  we  can  also  consider  finding  a  piece- wise  space  varying 

Hn  addition  to  using  the  approximation  (6),  we  tried  two  other  ap¬ 
proaches.  Firstly,  we  simply  substituted  Xnew,  Unew,  ginew  §2 new 

in  equation  (2)  and  equated  this  to  the  energy  computed  from  the  previ¬ 
ous  iteration,  so  as  to  get  the  value  for  Xnew-  Secondly,  we  tried  using 
the  value  of  the  residue  at  equilibrium  from  the  model  proposed  by  [16], 
in  place  of  the  second  term  in  the  equation  (2).  This  also  gives  a  simple 
expression  for  getting  the  next  A.  Both  of  these  methods  did  not  produce 
desirable  results,  especially  in  comparison  with  (7). 
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A,  following  work  done  in  [4]  for  the  TV  model  in  [13].  Prelim¬ 
inary  experiments  in  this  direction  show  that  although  there  is  a 
clear  improvement  in  the  energy,  visually  there  are  no  significant 
changes  when  compared  with  the  global  optima  computed  with  the 
technique  proposed  here.  Additional  results  in  this  direction  will 
be  reported  elsewhere. 
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Fig.  1.  Original  image  (top)  and  the  corresponding  decomposition. 
The  cartoon  image  (center)  has  some  texture  still  intact. 
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Fig.  2.  Plot  (left)  shows  how  our  method  fares  better  than  an  arbitrary  A.  The  cartoon  image  (center)  is  more  smooth  and  piecewise  constant 
than  the  corresponding  one  in  Figure  1. 


Fig.  3.  The  image  on  the  left  is  decomposed  with  arbitrary  A  to  produce  the  v  image  on  the  center  and  with  optimal  A  to  produce  the  one  on 
the  right.  The  v  image  in  the  center  still  contains  edges  at  the  place  of  light  contrast  change  (and  not  just  oscillations),  and  these  are  gone 
with  the  A  computed  with  our  approach.  . 


Fig.  4.  Illustration  of  the  application  of  our  method  for  inpainting,  following  [6].  The  image  in  the  center  is  reconstructed  using  arbitrary  A 
and  the  one  on  the  right  uses  our  method  for  selecting  A. 
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